Testing inference on Network Diffusion

Here, I will examine the utility of sampling and variational inference for inferring values from a simple network diffusion model on Erdos-Renyi random graphs. The primary aim of this document is to assess how well inference can scale as the network grows in size and as topology -- in this case, connection probability -- change.

Environment

First, check environment to ensure all packages needed are present and document their versions.

Model Setup

The first step in defining our model will be to initialise a graph on which to run the model. We do this using LightGraphs to generate a Erdos-Renyi random graph of size N.

The second step of the modelling process will be to define the ODE model. For network diffusion, this is given by:

$$ \frac{d\mathbf{u}}{dt} = -\rho \mathbf{L} \mathbf{u} $$

We can set this up as a julia function as follows:

To run a simulation, we set some initial conditions and define an ODEProblem using DifferentialEquations

And we can view the solution.

Inference

Now that we have a model, we generate some data and start to using Turing to perform inference. To do this, we should define a generative model.

Our data $\mathbf{y}$ is given by a normal distribution centered around our model $f(\mathbf{u0}, \rho)$ with variance $\sigma$.

$$\mathbf{y} = \mathcal{N}(f(\mathbf{u0}, \rho), \sigma)$$

and we assume our paramters are generated from the following distributions:

$$\sigma \approx \Gamma^{-1}(2, 3)$$

$$\rho \approx \mathcal{N}(5,10,[0,10])$$ $$\mathbf{u0} \approx \mathcal{N}(0,2,[0,1])$$

We can make this into a Turing model.

To fit this model, we first need to generate some data. We can then feed in our data and our model into the Turing model and begin to sample from it.

For now, we'll just use the data generated form our ODE solution above.

MCMC Sampling

We can now perform inference. First by initialising our fit function with synthetic data and our ODE problem. We can call initialise multiple chains to sampline in parallel -- here we use 10 chains.

Once the sampling has completed, we can plot the chains to visualise convergence and posterior distributions of parameters.

Summary

We can see from the plots and the chain summary that the chains converge and produce consistent estimates of the posterior distributions. Importantly, the posterior estimates closely correspond to the true model parameters.

With the ODE model and Turing model setup, we can begin to experiment with how inference is affected by changes to network topology and size.

How Does Inference Scale with a Growing Network

In this first experiment, we will test how well inference scales when we increase the size of the network used to simulate network diffusion.

We can do this by initalising a new network with size N and plugging this into our ODEProblem and Turing model.

For the benefit of speed, we will only use one chain for the following examples. In practice, using multiple chains should not take much longer and can be implemented easily, as above.

N = 10

Comment

The results are stable, with tight distributions around the mean despite taking relatively few samples from the posterior (1000).

N = 25

Comment

The results are still stable, with tight distributions. It is promising that the inference remains fast, despite increasing the number of nodes. This is likely due to simplicity of the model -- making auto diff easier and faster.

It is also encouraging that the results are accurate even though there are fewer informative data points available. That is, the diffusion time course toward equillibrium has decreased from 5 nodes to 10 to 25. This is most certainly due to the increased number of connections between nodes (higher mean degree). In the next case, there should be even fewer informative data points available.

N = 50

Comment

At N = 50, integrated over time span (0.0,1.0) and evaluated at 0.05 steps, inference using the NUTS sampler was slow and appears to not be converging. The ETA for inference continued to increase toward 9.5 hrs before I terminated. There are at least two likely reasons for this. 1) The model has reached a critical level of complexity such that the NUTS sampler is not able to efficienctly explore the posterior parameters space. 2) there is not sufficient data to be able for the parameters to be identified, resulting in non-convergence.

To test whether (1) or (2) is true, I will run a simulation integrating over a shorter time window and evaluating at 0.0125 steps.

Comment

The model converges for all parameters and produces highly accurate results.